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ABSTRACT 

The Student Operational Aggie Doppler Radar Project (SOAP) involved 95 undergraduates in a research and education 
program to better understand the climatology of storms in southeast Texas from 2006-2010. This paper describes the 
structure, components, and implementation of the 1-credit-hour research course, comparing first-year participants' 
experiences and career outcomes with students who were engaged in SOAP for multiple years. Groups of five or six students, 
led by a senior-level undergraduate and mentored by a graduate student and faculty advisor, performed several daily research 
tasks, including producing precipitation forecasts, archiving observations, and operating and analyzing data from an S-band 
Doppler radar for precipitation events on their assigned day. Anonymous surveys given to SOAP students at the end of each 
semester indicated that student confidence in performing most SOAP tasks exhibited statistically significant positive 
correlations with their interest and experience in doing them. In addition, students participating in SOAP for multiple years 
were significantly more confident in performing program tasks than single-year participants (with correlations increasing an 
average of 19%) and were more likely to obtain meteorology or science-related employment upon graduation (94% versus 
69%). First-year participants were significantly more likely to indicate that their interactions with undergraduate student 
leaders or peers were most beneficial, whereas interactions with the faculty advisor or graduate student mentors were equally 
or more important to returning students. Students were also more likely to consider research careers and matriculate to 
graduate programs as they participated longer in SOAP, suggesting research and education programs have a strong influence 
on students' career outcomes in addition to fostering positive self-efficacy. © 2013 National Association of Geoscience Teachers. 
[DOI: 10.5408/12-382.1] 
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INTRODUCTION 

Engaging undergraduates in research and educational 
field-based experiences in atmospheric science allows 
students to observe and analyze the environment outside 
the traditional classroom. These opportunities allow stu¬ 
dents to apply their existing knowledge while introducing 
them to higher-level concepts traditionally learned in more 
advanced courses. Undergraduate research has been regard¬ 
ed as an increasingly critical component of science education 
for several decades (Halstead, 1997; Hensel, 2012) that 
develops students personally and professionally and trans¬ 
forms faculty into "teaching scholars" (Lopatto, 2009). More 
than half (53%) of all science, technology, engineering, and 
mathematics (STEM) undergraduates participate in inde¬ 
pendent or mentored research activities (Russell, 2006). 
Studies consistently show that students clarify and confirm 
or modify their career goals after participating in authentic 
undergraduate research experiences (i.e., where students 
work on scientific research with practicing scientists), 
particularly with respect to their desire to attend graduate 
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school (e.g., Lopatto, 2004; Seymour et al., 2004; Gonzales- 
Espada and LaDue, 2006; Hunter et al., 2007; Russell et al., 
2007). Involving undergraduates in research experiences also 
increases the students' confidence in their scientific abilities, 
including perceived improvements in oral and written 
communication skills, computer and technical skills, and 
understanding of research processes (Kardash, 2000; Bauer 
and Bennett, 2003; Lopatto, 2004; Seymour et al., 2004). 

In addition to authentic research experiences, many 
undergraduates participate in field-based educational expe¬ 
riences, ranging from field campaigns to single courses 
involving larger numbers of students, that provide them with 
a better understanding of scientific careers, principles, and 
processes. The Rain in Cumulus over the Ocean (RICO) field 
campaign in 2005 involved undergraduates with graduate 
students from several institutions in a variety of educational 
activities, ranging from seminars and training sessions to 
hands-on experience with instrumentation, culminating 
with a student-led scientific mission involving the use of 
research aircraft (Rauber et al., 2007). On a smaller scale, 
university courses have been documented in the geosciences 
that involve 1-3 weeks of intensive field experiences that 
often require significant travel with pre- and post-trip 
assignments (e.g., Aitchison and Ali, 2007; Lathrop and 
Ebbett, 2006). Analogous courses in atmospheric science are 
beginning to exist, such as those that couple forecasting and 
storm chasing experiences with professional development 
activities (Godfrey et al., 2011). Local fieldwork experiences 
involving greater numbers of students have also been 
incorporated into introductory and upper-level geosciences 
courses at several universities (e.g., Comrie, 2000; Knapp et 
al., 2006; Elkins and Elkins, 2007; Morss and Zhang, 2008), 
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TABLE I: Number of weeks (out of 11) during which students interacted with the faculty advisor or one of the graduate student 
mentors during SOAP. 


Interactions With Faculty Advisor or Graduate Student Mentors 

2007 (n = 26) 

2008 (n = 29) 

2009 (n = 29) 

2010 (n = 30) 

8+ interactions 

15% 

62% 

55% 

67% 

5-7 interactions 

46% 

21% 

28% 

27% 

2-4 interactions 

35% 

17% 

14% 

6% 

0-1 interaction 

4% 

— 

3% 

— 


including a two-semester sequence in atmospheric science 
in which upper-division undergraduate and graduate stu¬ 
dents collected mobile radar data during fall semesters, with 
analysis occurring the following spring (Richardson et al., 
2008). All of these projects integrate research and education 
by incorporating a period of data collection and subsequent 
analysis activities that expose students to inquiry-based 
learning and critical thinking. 

This article evaluates undergraduate experiences and 
impacts from participating in the Student Operational Aggie 
Doppler Radar Project (SOAP), an atmospheric science 
research and education program at Texas A&M University 
(TAMU) that established a long-term storm climatology for 
southeast Texas while providing 95 students with applied 
experience. SOAP includes local fieldwork with authentic 
research tasks of collecting and analyzing data from the 
Aggie Doppler Radar (ADRAD) and supporting instruments 
as part of a research course offered each spring semester 
from 2006 to 2010. SOAP satisfies all seven principles for 
good practice in undergraduate education outlined by 
Chickering and Gamson (1999) by encouraging undergrad¬ 
uate student interaction with faculty and graduate students, 
cooperation among students, and active learning while 
providing prompt feedback, emphasizing time on task, 
communicating high expectations, and respecting diverse 
learning styles. However, SOAP is unique in that advanced 
undergraduate students led their peers (ranging from 
freshmen to seniors) in performing daily operational and 
research tasks, relying upon the faculty advisor and graduate 
student mentors for training materials, complicated logistics, 
and overall guidance. In addition to freeing time for faculty, 
peer-to-peer mentoring and instruction provide advanced 
students with the confidence-building experiences of leading 
and teaching students that promote scientific, student- 
centered cooperative learning (Rueckert, 2007; Judge et al., 
2012). Engaging upper- and lower-division students in 
SOAP also involved many undergraduates in research early 
in their college careers. This is a critical factor because 
students who perform research over multiple semesters tend 
to exhibit stronger outcomes and report greater benefits than 
those with shorter experiences (Bauer and Bennett, 2003; 
Russell, 2006). 

Sadler and McKinney (2010) emphasize that assessing 
the effectiveness of undergraduate research programs is 
essential for improving and advancing undergraduate 
science education. Evaluating SOAP is particularly important 
for the atmospheric sciences because most empirical studies 
of undergraduate research programs come from other STEM 
fields (Gonzales-Espada and LaDue, 2006) and focus on the 
program's research goals rather than the evaluation of 
participants' experiences and long-term impacts on career 
goals and outcomes. SOAP also appears to be unique 


because it integrates research and education through data 
collection and analysis with peer interactions and leadership 
involving many undergraduates instead of mentoring 
students through individual research projects. Therefore, 
this article investigates students' experiences and impacts 
from participating in SOAP by answering the following 
questions: 

• How is SOAP structured and how may similar 
programs be implemented elsewhere? 

• Do multiyear SOAP participants exhibit statistically 
significant differences from single-year participants in 
their confidence in performing SOAP tasks or career 
outcomes? 

• Which types of program interactions did students find 
most beneficial? 


SOAP PROGRAM DESCRIPTION AND 
IMPLEMENTATION 

Course Structure 

SOAP involved 25-30 undergraduates at TAMU each 
spring semester between 2006 and 2010, providing 95 
unique students with forecasting and research practice. 
Students earned 1 research credit hour graded on a pass/fail 
basis by the faculty advisor, who was the principal 
investigator of the supporting National Science Foundation 
(NSF) Faculty Early Career Development (CAREER) grant 
and the second author of this paper (the first and third 
authors were graduate student mentors during all 5 y and 
the last 3 y of SOAP, respectively). Participants gained 
forecasting and research practice by predicting, observing, 
and analyzing a variety of precipitating systems while 
collecting data and providing useful results toward the 
broader NSF grant's research goals. SOAP students orga¬ 
nized themselves into daily groups of five or six individuals 
who were responsible for producing a precipitation forecast, 
operating ADRAD if precipitation was in the radar domain 
on their group's assigned day, and performing several other 
tasks discussed in the next section. Each daily group was led 
by an upper-level undergraduate with past SOAP experience 
who was directly mentored by one of three graduate 
students and the faculty advisor. The graduate student 
mentors and faculty advisor also made a conscientious effort 
to regularly interact with SOAP participants, with most 
students reporting eight or more interactions with these 
mentors out of the 11 operational weeks of the program 
during the last 3 y of SOAP (Table I). 

Organizational and training meetings were held during 
the second and third weeks of the semester after students 
finalized their course schedules, followed by 11 weeks of 
operations and a wrap-up meeting during the last week of 
the semester. Although some programmatic elements varied 
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TABLE II: Sample course schedule given in the spring 2010 SOAP syllabus. 


Week 

Dates 

Meetings and Activities 

1 

1/18-1/22 

No SOAP activities this week 

2 

1/25-1/29 

Organizational Meeting on Monday 1/25 at 7 p.m. 

3 

2/1-2/5 

SOAP Operations Training Meeting on Monday 2/1 at 7 p.m. 

Radar Operations and Cloud Observations Training on shift day 

4 

2/8-2/12 

SOAP Operations Week 1; Radar learning module 1 (radar startup) 

5 

2/15-2/19 

SOAP Operations Week 2; FX-Net learning module 1 (fronts and isentropic analysis) 

6 

2/22-2/26 

SOAP Operations Week 3 

7 

3/1-3/5 

SOAP Operations Week 4; Radar learning module 2 (image analysis) 

8 

3/8-3/12 

SOAP Operations Week 5 


3/15-3/19 

NO CLASSES OR SOAP OPERATIONS ALL WEEK (Spring Break) 

9 

3/22-3/26 

SOAP Operations Week 6; FX-Net learning module 2 (sounding analysis) 

Intensive Operation Period (IOP) with Sounding Launches Begins (Week 1) 

Sounding Launch Training on shift day; meet at 5 p.m. (weather permitting) 

10 

3/29-4/2 

SOAP Operations Week 7 and IOP Week 2 

11 

4/5—4/9 

SOAP Operations Week 8 and IOP Week 3 

12 

4/12-4/16 

SOAP Operations Week 9 and IOP Week 4; FX-Net learning module 3 (Q-vectors, ageostrophic 
motions, and local circulations) 

13 

4/19-4/23 

SOAP Operations Week 10 and IOP Week 5; Radar case studies 

14 

4/26-4/30 

SOAP Operations Week 11 and IOP Week 6; Radar case studies 

15 

513-517 

Wrap-up Meeting on Wednesday 5/5 at 7 p.m. 

16 

5/10-5/14 

No SOAP activities this week 


each year (as discussed in subsequent paragraphs), the 
sample schedule from the spring 2010 SOAP syllabus 
provided in Table II outlines the general course structure 
followed every year except for 2006, when only 7 weeks of 
operations occurred. The first mandatory evening meeting 
occurred during the second week of the semester and 
provided a brief orientation to the program while allowing 
students to organize themselves into daily groups and set 
their daily meeting times. During the third week of the 
semester, each group met during their regular meeting time 
for training on radar operations and cloud identification, in 
addition to a second mandatory evening session covering 
forecasting and new program elements. Regular daily SOAP 
operations began during the fourth week and continued 
until the last week of the semester, when a final mandatory 
evening wrap-up meeting was held to summarize, recog¬ 
nize, and evaluate the progress made during the semester. 
Each group also presented their most interesting case study 
and competed in a wildly popular Jeopardy!- style game that 
tested the overall content knowledge students had learned 
by completing daily tasks described in the following section. 

Description of Daily SOAP Tasks 

Students were expected to be on call for their group's 
day from 8 a.m. until 10 p.m. to operate the radar and 
perform several other tasks, many of which were part of 
SOAP every year. Groups met in the morning and afternoon 
for 1-2 h to perform their daily responsibilities in a lab space 
designated for SOAP, meeting longer if rain was in the area. 
Daily tasks incorporated into every year of SOAP included 
the following: 


• Forecasting precipitation and writing the daily forecast 
discussion during the morning hours, with a corre¬ 
sponding afternoon update. Each shift analyzed 
national satellite and radar imagery and upper-air 
maps before focusing regionally on southeast Texas. 
Students then posted a forecast discussion to the 
SOAP Web site (TAMU, 2009), indicating the 
likelihood that precipitation would occur before 8 
a.m. the next day and over the course of the next 3 d. 
In addition to serving a practical operational purpose 
for SOAP, writing forecast discussions has been 
shown to improve student performance and confi¬ 
dence in the precipitation forecasts they produce 
(Market, 2006). 

• Identifying cloud types according to the World Mete¬ 
orological Organization (WMO) synoptic code 
(WMO, 1974) using the 10 classifications (including 
no cloud) for low, middle, and high clouds. Students 
took eight panoramic snapshots of the sky during the 
morning and afternoon from the 15th-floor observa¬ 
tory in the O&M Building, posting these images to the 
SOAP Web site in real time. 

• Operating the radar on the roof of the O&M Building 
when rain was within 150 km between 8 a.m. and 10 
p.m. local time. Students chose appropriate scan 
strategies from a preset selection depending on the 
depth of the storm system and monitored the radar to 
ensure data were being continuously collected. Group 
leaders organized their group members in covering 
the radar throughout the day and coordinated with 
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their graduate student mentor and the next day's 
group leader if overnight operations were necessary. 

• Analyzing radar data in real time and from past cases 
using Sigmet's Interactive Radar Information System 
(IRIS). Each group also created representative images 
with IRIS and wrote and posted a precipitation 
summary to the SOAP Web site for each event that 
occurred on their day, presenting one case study in 
detail at the wrap-up meeting. 

In addition to these daily tasks that students performed 
each year of SOAP, several activities were implemented for a 
few years as the program matured. The following tasks were 
included: 

• Analyzing online archives for the radar climatology 
(2007-2008) to classify storms' dynamical forcing 
and structures using archived surface and upper-air 
maps, satellite imagery, and Next-Generation Radar 
(NEXRAD) imagery. Advanced SOAP students vetted 
these storm classifications the following fall semester 
and performed preliminary analyses for the rainfall 
climatology presented in Hopper (2011). 

• Using FX-Net software to forecast (2008-2010), in 
addition to the Web-based resources used during 
the first 2 y of SOAP. FX-Net simulates the National 
Weather Service's (NWS's) Advanced Weather Inter¬ 
active Processing System (AWIPS) graphical user 
interface, permitting students to create analysis 
products in addition to default products specifically 
created for SOAP. 

• Learning modules for FX-Net and ADRAD (2008-2010) 
that taught students how to perform specific tasks or 
learn methods of analysis for forecasting using FX- 
Net or radar data in IRIS. Groups typically completed 
four to six learning modules a semester on relatively 
slow days when precipitation was not in the area. 

• Recording and submitting rain gauge data (2009-2010) 
daily between 7 and 9 a.m. local time on the 
Community Collaborative Rain, Hail, and Snow 
Network (CoCoRaHS) Web site (Colorado Climate 
Center, 2011) for gauges sited by each student, 
typically at that student's residence. 

• Creating a quantitative precipitation forecast (2009-2010) 
based on past rainfall totals for their predicted storm 
type and structure, model output, and the Storm 
Prediction Center's short-range ensemble forecasts. 
Students forecasted the timing of rainfall and made a 
mean, minimum, and maximum quantitative precip¬ 
itation forecast that they verified the following week 
using data from approximately 30 SOAP CoCoRaHS 
gauges sited within 10 km of ADRAD. 

• Performing sounding launches (2008 and 2010) from the 
17th green of the TAMU Golf Course. All groups 
released two radiosondes at 0000 UTC during the last 
2 weeks of April 2008, whereas each group was given 
three radiosondes to release at 0000, 1200, or 1800 
UTC on days of their choice during the last 5 weeks of 
SOAP in 2010. 

Implementation Strategies 

Although SOAP was originally designed as an under¬ 
graduate research and education program in atmospheric 


science for a research university with very high research 
activity (as defined by the Carnegie Classification of 
Institutions of Higher Education [Carnegie Foundation for 
the Advancement of Teaching, 2013]), the general course 
structure could be adapted to suit the needs of institutions 
with more limited resources or other degree programs in 
geosciences or STEM fields. In addition to peer mentoring, 
we recommend that research and education programs 
include an initial set of training sessions or modules at the 
beginning, advanced training and regular data collection and 
analysis tasks throughout the project, and a final product 
synthesizing each group's work at the project's end. Specific 
implementation strategies will depend on individual re¬ 
sources and objectives, but the following practices are 
recommended based on experiences in implementing and 
managing SOAP: 

• Recognize what is within the scope of your resources and 
time. In agreement with White et al. (2008), imple¬ 
menting and running SOAP was initially time 
consuming for the faculty advisor and would have 
been difficult to maintain without one part-time and 
two full-time graduate students that spent 5-15 h 
each week on SOAP during the semester. Therefore, 
faculty at primarily undergraduate institutions (PUIs) 
may consider departmental or interdisciplinary col¬ 
laborations with other faculty to share in mentoring 
undergraduates and handling logistics or scale back 
the size and scope of their research experiences. The 
first author has accomplished this at a PUI by using 
existing laboratory or seminar courses that are part of 
his regular teaching load to mentor groups of three to 
five students through semester-long research projects 
that satisfy course objectives. 

• Encourage all levels of students to participate by recruiting 
deliberately, facilitating enrollment, and including data 
collection and analysis tasks involving all students. The 
faculty advisor and graduate student mentors recruit¬ 
ed participants by advertising SOAP at student 
organization meetings and major courses, highlight¬ 
ing the unique opportunity to gain practical experi¬ 
ence in radar operation and analysis; more than half 
of the participants identified this as a primary reason 
for joining SOAP. Ten of the 30 seats were reserved 
for lower-division students by cross-listing SOAP as a 
sophomore- and senior-level course at TAMU, but 
other institutions may need a different strategy. The 
faculty advisor recruited senior undergraduates with 
leadership abilities and radar experience to serve as 
group leaders the first semester, whereas subsequent 
leaders had SOAP experience. Once the program was 
established, returning SOAP participants indirectly 
recruited by sharing their experiences with potential 
students. One student even wrote that he was 
"referred to SOAP by a friend that had no experience 
in SOAP" and thought it would introduce him to 
"running a radar and having an extremely expensive 
machine at (his) disposal." 

• Establish an organizational framework with clear expec¬ 
tations, but give students autonomy in performing their 
research tasks and embrace flexibility. Communicating 
expectations to the participants within the syllabus 
and during the first meeting is vital so that all students 
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understand that their leaders and the faculty advisor 
will hold them accountable if they abuse the 
autonomy they are given. Students who did not meet 
basic expectations or who had more than one 
unexcused absence were required to complete addi¬ 
tional project work to receive a passing grade. In 
addition, establishing a course structure that is flexible 
enough to accommodate alternative plans is impor¬ 
tant in case things do not go as expected, such as an 
extended period when AD RAD was inoperable during 
several weeks of SOAP in 2008. Proactively planning 
for these events by having possible contingency plans 
(e.g., launching soundings in 2008 instead of radar 
operations) contributed to SOAP's success by provid¬ 
ing useful research data while still generating 
excitement and benefitting students educationally. 

• Encourage students to pursue additional research oppor¬ 
tunities. Thirteen SOAP participants performed eight 
authentic research projects under the direction of the 
faculty advisor or a graduate student mentor during 
the fall 2007, 2008, and 2009 semesters, culminating 
with presenting their work at the American Meteo¬ 
rological Society's Annual Student Conference and 
TAMU Student Research Week. Most of these "Fall 
SOAP" projects incorporated data collected by spring 
participants or furthered the analyses performed by 
students during the regular program. Some of these 
advanced projects also influenced the research and 
data collection objectives of SOAP between 2008 and 
2010 when many of the group leaders had participat¬ 
ed in fall SOAP. 

• Utilize student feedback when implementing improve¬ 
ments or new tasks. Monthly meetings among the 
group leaders, graduate student mentors, and faculty 
advisor were held during December-April for brain¬ 
storming, training, and feedback, particularly on new 
research objectives and activities. Improvements 
implemented during the semester were driven by 
suggestions from group leaders during these meet¬ 
ings, whereas changes made for future semesters 
primarily arose from comments made by participants 
who evaluated their experiences at the end of the 
semester. Several key educational findings from these 
student surveys are discussed in the following section. 

SURVEY ASSESSMENT METHODS 

Anonymous surveys approved by the Institutional 
Review Board at TAMU were administered at the end of 
the semester during the last 4 y of the program (2007-2010). 
The undergraduate meteorology program at TAMU had 
approximately 140-150 majors over this period, meaning 
about 20% of the department's majors were involved in 
SOAP at one time. Excluding 13 students who only 
participated in SOAP during 2006, 77 of the 82 remaining 
participants completed 114 surveys, 28 of whom completed 
surveys in multiple years. Survey items were generally the 
same each year with only slight variations. The average 
survey response rate was 95% (114 of 120) over the 4-y 
period ranging from 87% in 2007 to 100% in 2010. 

SOAP involved 41 males and 41 females from 2007- 
2010, 18% (8 males and 7 females) of which were from 
underrepresented minority groups (primarily Hispanic or 


Latino), similar to the undergraduate student body at TAMU 
during this period. SOAP involved a higher percentage of 
females (50%) than typically enrolled in most atmospheric or 
earth science degree programs across the United States 
(Charlevoix, 2010), including the meteorology program at 
TAMU (58% male and 42% female). Seven non-meteorology 
majors (four environmental geosciences, two civil engineer¬ 
ing, and one geology) and seven nontraditional students also 
participated in SOAP. Seniors, juniors, and lower-division 
students each accounted for 30%-40% of the students. 

All questions on the anonymous, non-longitudinal 
survey were optional. Participants only identified their 
classification (upper or lower) and years of SOAP experience 
without identifying individual demographics such as age, 
gender, or ethnicity to ensure student privacy. In addition, 
students indicated which career goals they were considering 
upon graduation, including forecasting, research, and 
broadcast meteorology. Qualitative responses were solicited 
through open-ended questions, some of which prompted 
participants to identify how specific components of the 
program could be improved or what they would like to see 
added. Others asked students to indicate whether interac¬ 
tions with their peers, group leaders, graduate student 
mentors, faculty advisor, or a combination of these was most 
beneficial. Quantitative responses prompted students to 
specify how often they completed many of the SOAP tasks 
discussed in the previous section (i.e., experience), in 
addition to rating their confidence and interest in performing 
each task using Likert-type items rated on a discrete five- 
point scale. The exact wording of the question, items, and 
response scale descriptors used for each scale in the survey is 
provided in Table III. Although this paper focuses on 
quantitative results, qualitative responses are used in 
providing supporting anecdotes and in determining which 
student interactions were most helpful and why. 

Although some studies (e.g., Norman, 2010) defend the 
use of parametric statistical measures for Likert (1932) scales, 
nonparametric statistical measures are less objectionable for 
analyzing ordinal data, including Likert scales and individual 
Likert-type items. This is particularly true for datasets that 
are small or do not exhibit normal distributions 0amieson, 
2004). Therefore, nonparametric statistics are typically used 
in this article, beginning with Table III that shows the 
median and mode student responses for confidence, interest, 
and experience for individual SOAP tasks. Students exhib¬ 
ited high confidence and interest in performing most 
program tasks each year, except for creating quantitative 
precipitation forecasts and completing learning modules 
when they were first implemented. Participants reported 
similar experience levels in operating ADRAD and analyzing 
its data each year but were least confident in 2008, when the 
radar was inoperable for several weeks. Aside from increased 
participation in sounding launches during 2010 relative to 
2008, students reported similar levels of experience each year 
for other tasks. 

Internal consistency of the confidence, interest, and 
experience scales were measured using the Cronbach's alpha 
estimate of the reliability coefficient, or the squared 
correlation between the observed and the true values of a 
variable (Table IV). Cronbach's alpha values indirectly 
indicate how well a set of items measures a single underlying 
construct, ranging from 0-1.0, with 1.0 expressing perfect 
reliability. Alpha values exceeding 0.7 are generally regarded 
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TABLE III: Median and mode responses for items comprising the confidence, interest, and experience Likert scales. Modes are 
given in parentheses when they differ from their respective medians. Items without values refer to activities not performed by 
students during that year's program. 1 



2007 (n = 26) 

2008 (n = 29) 

2009 (n = 29) 

2010 (n = 30) 

CONFIDENCE SCALE: How CONFIDENT are you in performing the following activities? 2 

(1) Operating the radar (ADRAD) 

4.5 (5) 

3 

4 

4 

(2) Analyzing radar data (cross-sections, precipitation summary) 

4 

3 

4 

4 (5) 

(3) Forecasting and writing the daily forecast discussion 

4(5) 

4 

4 

5 

(4) Utilizing the FX-Net software to forecast 

— 

4 

4(5) 

4 

(5) Creating a quantitative precipitation forecast (QPF) 

— 

— 

3 

3 

(6) Performing sounding launches 

— 

4 

— 

4.5 (5) 

(7) Learning modules for radar (ADRAD) and/or FX-Net 

— 

4 (3) 

4 

4 

INTEREST SCALE: How much did you LIKE performing the following activities? 2 

(1) Operating the radar 

4.5 (5) 

5 

5 

4 

(2) Analyzing radar data 

4 

4 

4 (5) 

4 (5) 

(3) Forecasting and writing discussion 

4(5) 

4 

4(5) 

4 

(4) Utilizing software to forecast 

— 

4 

4 

4 

(5) Creating a precipitation forecast 

— 

— 

3 

3 

(6) Performing sounding launches 

— 

5 

— 

5 

(7) Learning modules 

— 

3 

4 (3) 

4 (5) 

EXPERIENCE SCALE: Approximately how many times/weeks (out of 11) did you perform each activity this semester? 3 

(1) Operating the radar 

3 

3 

3 

3 

(2) Analyzing radar data 

3 

3 

3 

3 (4) 

(3) Forecasting and writing discussion 

4 (3) 

4 

4 

4 (5) 

(4) Utilizing software to forecast 

— 

4 

4 

4 

(5) Creating a precipitation forecast 

— 

— 

3 

3 (2) 

(6) Performing sounding launches 

— 

2 

— 

4 

(7) Learning modules 

— 

3 

4 

4 


'Four participants in 2007, one in 2008, and one in 2009 did not complete the survey. 

Participants responded using a five-point Likert format (1 = no confidence, 3 = neutral, and 5 = very confident for confidence scale; 1 = disliked, 3 = neutral, 
and 5 = liked for interest scale). 

Participants responded using a five-point Likert format (1 = no experience and 5 = very experienced). Ranges of the number weeks during which participants 
performed each task were used in the place of discrete values on the survey as follows: 1 = 0, 2 = 1, 3 = 2-4, 4 = 5-7, and 5 = 8+ for Items 1-4 and 1 = 0, 2 = 
1, 3 = 2, 4 = 3, and 5 = 4+ for Items 5-7. 


as acceptable, whereas values below 0.5 are unacceptable 
(George and Mallery, 2003). Three SOAP tasks (identifying 
cloud types, recording and submitting rain gauge data, and 
analyzing online archives for the radar climatology) were 
removed from the analysis due to the confidence, interest, or 
experience scales exhibiting poor statistical relationships 


(i.e., low factor loadings) in multiple years. For example, 
virtually all participants performed daily rain gauge obser¬ 
vations during 2009 and 2010 and rated this activity as "5" 
for both experience and confidence, therefore reducing the 
overall reliability of the metric. The reliability of the 
confidence and experience scales (for SOAP as a whole) 


TABLE IV: Internal consistency of survey instrument scales by year. 1 


Likert Scale 2 

2007 a 

2008 a 

2009 a 

2010 a 

Cronbach's alpha for TOTAL CONFIDENCE scale 

0.76 

0.67 

0.51 

0.80 

Cronbach's alpha for TOTAL INTEREST scale 

0.49 

0.71 

0.61 

0.72 

Cronbach's alpha for TOTAL EXPERIENCE scale 

0.54 

0.67 

0.64 

0.54 

Cronbach's alpha for RADAR CONFIDENCE subscale 

0.84 

0.72 

0.71 

0.74 

Cronbach's alpha for RADAR INTEREST subscale 

0.53 

0.76 

0.65 

0.64 

Cronbach's alpha for RADAR EXPERIENCE subscale 

0.57 

0.82 

0.58 

0.76 


'Bolded Cronbach's alpha values are regarded as acceptable, whereas italicized values are "poor" or "unacceptable" (George and Mallery, 2003). 
2 Total scales use all items given in Table III, whereas the radar scales only use Items 1 and 2. 
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TABLE V: Confidence correlation coefficients with experience and interest for SOAP participants by year from 2008-2010. 1 ' 2 


Activity/Scale 

Confidence and Experience 

Confidence and Interest 

2008 

2009 

2010 

2008 

2009 

2010 

(1) Operating the radar 

0.37 

0.09 

0.33 

0.38 

0.45 

0.48 

(2) Analyzing radar data 

0.59 

-0.29 

0.29 

0.35 

0.06 

0.54 

(3) Forecasting and writing discussion 

0.54 

0.27 

0.20 

0.49 

0.68 

0.51 

(4) Utilizing software to forecast 

0.72 

0.52 

0.39 

0.54 

0.60 

0.42 

(5) Creating a precipitation forecast 

— 

0.31 

0.46 

— 

0.73 

0.59 

(6) Performing sounding launches 

0.71 

— 

0.56 

0.40 

— 

0.57 

(7) Learning modules 

0.51 

0.33 

0.53 

0.39 

0.72 

0.47 

TOTAL SCALE 

0.54 

0.05 

0.34 

0.48 

0.38 

0.57 

RADAR SUBSCALE (items 1 and 2 only) 

0.52 

-0.24 

0.27 

0.44 

0.35 

0.54 


'Bolded values are statistically significant at the 95% confidence level (p < 0.05 for two-tailed test). 

2 Pearson's correlation coefficient between ranks is used to calculate the Spearman rank correlation coefficients because of tied ranks (Myers et al., 2010). 
Survey results from 2007 are not presented because of their low reliability. 


increased when removing this task, whereas the reliability of 
the interest scales were largely unaffected due to having 
more variability among student responses. 

Results are presented for an overall SOAP Likert scale 
created by summing together the seven tasks in Table III and 
a radar subscale including responses for operating ADRAD 
and analyzing the collected radar data. In general, the 
confidence and interest scales are more reliable than the 
experience scale, whose responses were converted to a five- 
point scale based on ranges of weeks students performed 
specific tasks (see footnote in Table III). Although a five- 
point experience scale was used to better match the 
confidence and interest scales and simplify selections for 
the survey participants, assigning discrete values to contin¬ 
uous data may have introduced errors that partially account 
for the experience scale's lower reliability. Although results 
presented for scales exhibiting poor (0.5-0.6) or questionable 
(0.6-0.7) reliabilities should be considered carefully, all scales 
and radar subscales exceed the unacceptable threshold 
except interest responses during 2007. 

SURVEY ASSESSMENT RESULTS 
Correlating Confidence in Performing SOAP Tasks 
With Experience and Interest 

Empirical analyses of quantitative survey elements 
focused on analyzing student-reported self-efficacy, or 
confidence in performing specific SOAP tasks. Regardless 
of whether they are accurate or misguided, students base 
their judgments of self-efficacy on first-hand mastery 
experiences, comparing their abilities to others through 
vicarious experiences, social persuasion through verbal 
praise or discouragement from others, and reacting to 
physiological states such as fear and stress (Bandura, 
1986). Defined as the "belief in one's capabilities to organize 
and execute the courses of action required to manage 
prospective situations" (Bandura, 1995), self-efficacy has also 
been shown to affect student interests, career choices, and 
performance in educational and career pursuits (Lent et al., 
1994). Although the lack of a longitudinal set of surveys 
precludes direct investigation of the latter two assertions, a 
construct validity for the survey is established by correlating 
students' confidence in executing SOAP tasks (self-efficacy) 


with their breadth of experience and interest in performing 
them. 

Table V shows that student-reported confidence exhibits 
positive correlations with their experience and interest in 
performing most SOAP tasks from 2008-2010, with corre¬ 
lations not presented for 2007 because the interest and 
experience scales demonstrate unacceptable or poor reli¬ 
ability. Correlations between confidence and interest are 
statistically significant for all SOAP tasks except analyzing 
radar data during 2008 and 2009. Correlations between 
confidence and experience are typically less significant, 
particularly for forecasting, operating ADRAD, and analyz¬ 
ing radar data, the latter of which demonstrates a negative 
correlation during 2009. However, drawing conclusions from 
the 2009 data is difficult due to the confidence scale's poor 
reliability (cf. Table IV)- Nevertheless, the high correlations 
between confidence and interest for most SOAP tasks each 
year suggest the surveys are valid and imply that students' 
confidence in performing SOAP tasks is more strongly 
related to interest than experience. 

Comparing Confidence and Career Outcomes for 
Single- and Multiyear Students 

Previous studies indicate that undergraduates involved 
in research for multiple semesters report greater perceived 
benefits, including stronger career outcomes and higher 
levels of self-efficacy (Bauer and Bennett, 2003; Russell, 
2006). Therefore, statistical comparisons between "multi¬ 
year" students with prior SOAP experience and "first-year" 
students without prior SOAP experience are performed 
using the Mann-Whitney Li-test, which is the nonparametric 
version of the Student's t-test that is most appropriate for 
ordinal Likert scale data. The total confidence scale response 
distributions shown in Fig. 1 indicate that multiyear 
participants are significantly more confident than first-year 
participants in performing all SOAP tasks as a whole during 
each year of the program, with their mean confidence scores 
increasing 19% on average, ranging from 11% in 2009 to 
28% in 2007. These confidence variations are not associated 
with significant differences between the experience and the 
interest reported by both groups each year (not shown). 

Although the confidence variations are not statistically 
significant for each task, Fig. 2 shows that multiyear 
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Total SOAP Confidence Scale Response Distributions 

First-Year Participants Multi-Year Participants 

2007: U = 38.0, p = 0.02 



N=15 

Mean Rank: 10.5 
Mean Response (Sum): 3.2 (9.7) 

8 6 4 2 


N=11 
Mean Rank: 17.5 

Mean Response (Sum): 4.1 (12.4) 
2 4 6 8 


2008: U = 16.5, p = 0.001 


N=14 
Mean Rank: 8.7 

Mean Response (Sum): 3.3 (19.9) 

8 6 4 2 


N=15 
Mean Rank: 20.9 

Mean Response (Sum): 4.05 (24.3) 
2 4 6 8 


2009: U = 47.0, p = 0.01 



N=16 

Mean Rank: 11.4 

Mean Response (Sum): 3.7 (22.4) 

8 6 4 2 


N=13 

Mean Rank: 19.4 

Mean Response (Sum): 4.1 (24.8) 
2 4 6 8 


2010: U = 47.0, p = 0.03 



N=21 
Mean Rank: 13.2 

Mean Response (Sum): 3.8 (26.3) 

8 6 4 2 
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FIGURE 1: Total confidence scale mean response distributions (of the items in Table III that students performed each 
year) for first-year and multiyear SOAP participants during 2007-2010. Mann-Whitney U -test statistics, p values 
calculated using SPSS software, and mean ranks, item responses, and summated scores using three items in 2007, six 
items in 2008 and 2009, and seven items in 2010 are also provided. Participants responded using a five-point Likert 
format (1 = no confidence, 3 = neutral, and 5 = very confident). 


participants are significantly more confident in operating 
ADRAD and analyzing radar data than first-year SOAP 
members during each year, with radar subscale mean 
confidence scores increasing an average of 25%. These 
confidence increases are most exaggerated during 2008 
(36% increase) when ADRAD was inoperable for several 


weeks due to technical issues, causing returning SOAP 
students to have greater cumulative levels of radar 
experience that likely resulted in their higher confidence 
levels. Similarly, returning SOAP students are only 16% 
more confident than first-year participants in performing 
radar tasks during 2009 because SOAP students in their 
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Radar SOAP Confidence Scale Response Distributions 

First-Year Participants Multi-Year Participants 

2007: U = 44.0, p = 0.04 
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2008: U = 31.0, p = 0.001 



N=14 

Mean Rank: 9.7 H 
Mean Response (Sum): 2.75 (5.5) 


N=15 
Mean Rank: 19.9 

Mean Response (Sum): 3.75 (7.5) 


3 113 

2009: U = 50.5, p = 0.02 
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FIGURE 2: Radar tasks confidence scale mean response distributions (of Items 1 and 2 in Table III) for first-year and 
multiyear SOAP participants during 2007-2010. Mann-Whitney ll-test statistics, p values calculated using SPSS 
software, and mean ranks, item responses, and summated scores fusing both items each year are also provided. 
Participants responded using a five-point Likert format (1 = no confidence, 3 = neutral, and 5 = very confident). 


second year of the program did not have as much prior 
radar experience to draw from as their counterparts during 
other years. All of these results suggest that additional 
student experience with radar operation and analysis is 
critical to increasing their radar self-efficacy and their 
confidence in operating and analyzing data collected from 
specialized research instrumentation. 


Participants also indicated their career goals after 
finishing SOAP, with 65% expressing interest in forecasting 
occupations, compared to 39% and 34% for research and 
broadcast meteorology professions, respectively (Table VI). 
Although many first- and second-year SOAP participants 
selected multiple career goals, students involved in SOAP for 
3-4 y typically narrowed their career goals to one 
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TABLE VI: Stated career interests of SOAP participants during 2007-2010. 1 


Career Interests of SOAP Participants per Years Participation 

1 y (m = 66) 

2 y (n = 32) 

3-4 y (n = 16) 

All (n = 114) 

Weather forecasting (e.g., private sector, NWS, military) 

68% 

69% 

44% 

65% 

Research (i.e., graduate school) 

36% 

41% 

44% 

39% 

Broadcast meteorologist 

35% 

41% 

19% 

34% 


1 Students were allowed to choose multiple careers or none of those listed. In addition, some multiyear participants are reflected in multiple student categories 
depending on which year they completed the survey. 


occupation, indicating large decreases in forecasting and 
broadcast meteorology career goals with slight increases in 
attending graduate school and pursuing a career in research. 
In addition, participants interested in research careers 
generally exhibited slightly greater confidence and interest 
in performing SOAP tasks than those not interested in 
research careers, but these trends are only statistically 
significant for confidence during 2010 and are contradicted 
by nonsignificant trends of the opposite effect in 2008 (not 
shown). Although students interested in pursuing research 
careers were slightly more confident and interested in 
writing forecast discussions than those not interested in 
research careers during 2010, students not interested in 
research careers were significantly more confident in writing 
forecast discussions and enjoyed doing so more during 2008 
and 2009 (not shown). Most of these students expressed an 
interest in forecasting careers, providing support for Lent et 
al.'s (1994) finding that self-efficacy affects career goals while 
suggesting that not all program tasks were strictly geared 
toward satisfying students interested in research. 

Career outcomes of 74 bachelor's degree recipients that 
participated in SOAP during 2007-2010 were also deter¬ 
mined through social networking sites (e.g., Facebook and 
Linkedln) and other personal contacts. Similar to the 
national career statistics presented by Knox (2008) for 624 
meteorology bachelor's degree recipients during 2003-2005, 
34% of SOAP graduates obtained forecasting careers (39% if 
broadcast meteorologists are included) and 29% matriculat¬ 
ed to graduate programs, primarily in atmospheric sciences 
(Table VII). Multiyear SOAP participants were twice as likely 
to obtain forecasting positions after graduation as single¬ 
year participants, a result that is statically significant (p < 
0.05; Z-test for comparing two proportions). In agreement 
with Bauer and Bennett (2003) and Russell (2006), graduates 
participating in SOAP for 2 y were 39% more likely to attend 
graduate school and pursue research careers than were those 
involved in SOAP for 1 y; those participating for 3-4 y were 


65% more likely. In addition, 94% of graduates that 
participated in SOAP at least 2 y obtained meteorology or 
science-related employment compared to 69% of single-year 
SOAP participants, differences that are statistically signifi¬ 
cant (p < 0.01; Z-test for comparing two proportions). 

Finally, students were not surveyed on their initial career 
goals or their experience, confidence, or interest in 
performing tasks similar to those in SOAP before the 
program each year. Capturing students' thinking on careers 
before participating in SOAP would have been beneficial, 
because undergraduates are in a phase of their lives during 
which they are continually learning more about career 
opportunities and assessing their interest and ability in 
obtaining jobs they may desire. Environmental and involve¬ 
ment variables outside of SOAP may be responsible for 
some of the participants' changes in career goals and 
outcomes, including their parents' occupations, socioeco¬ 
nomic status, standardized test scores, prior coursework, 
participation in other activities (or internships), and peer 
groups (Astin, 1993). Reliable data on career goals and 
outcomes for meteorology students who were not involved 
in SOAP were also not available. However, many students 
credited SOAP with motivating them to pursue and obtain 
research assistantships, forecasting, or other scientific jobs 
upon graduation on departmental exit surveys administered 
to meteorology students after 2009. Nevertheless, our results 
comparing single- and multiyear participants in SOAP 
suggest that repeated exposure to the same type of applied 
research and educational experiences outside the traditional 
classroom increases the likelihood that students will remain 
in science after graduation, which is a retention issue that 
should be considered as important as graduation and 
university retention rates. 

Comparing the Importance of Interpersonal 
Interactions for Single- and Multiyear Students 

Involvement with and the nature of a student's peer 
group has been shown to influence student growth, 


TABLE VII: Career outcomes of bachelor degree recipients as of summer 2012 that participated in SOAP during 2007-2010. 1 


Career Choices of SOAP Graduates per Years Participation 

1 y (n = 39) 

2 y (n = 22) 

3-4 y (n = 13) 

All (n = 74) 

METEOROLOGY OR SCIENCE OCCUPATIONS 

69% 

91% 

100% 

81% 

Weather forecasting (e.g., private sector, NWS, military) 

23% 

41% 

54% 

34% 

Research (i.e., graduate school) 

23% 

32% 

38% 

28% 

Other science occupations (e.g., analyst, technician, teacher) 

18% 

13.5% 

— 

14% 

Broadcast meteorologist 

5% 

4.5% 

8% 

5% 

NONSCIENCE OR UNKNOWN OCCUPATIONS 

31% 

9% 

— 

19% 

Nonscience occupations 

15.5% 

4.5% 

— 

9.5% 

Unknown occupations 

15.5% 

4.5% 

— 

9.5% 


1 Career outcomes of 13 BS degree recipients that only participated in SOAP during 2006, 5 students whose degrees are still in progress, and 3 students who 
did not graduate are excluded. The table includes 67 meteorology, 4 environmental geosciences, 2 civil engineering, and 1 geology degree recipient. 
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development, and academic success most during college, 
followed by faculty interactions, particularly those connected 
with academic experiences outside the classroom (Astin, 
1993). Several students indicated they joined SOAP to 
interact and work with other meteorology majors or the 
faculty advisor or because peers or the faculty advisor 
recommended the program. Indeed, synergy between peers 
and undergraduate group leaders in performing program 
tasks and abundant informal student interactions with the 
faculty advisor or graduate student mentors (cf. Table I) are 
both critical components of SOAP that undoubtedly 
contributed to student learning. Although 41% of students 
said their most beneficial interactions were with the faculty 
advisor or graduate student mentors (Table VIII), first-year 
participants were significantly more likely to indicate that 
their interactions with undergraduate student leaders or 
peers were most beneficial {p < 0.01; Z-test for comparing 
two proportions). Several students also stated that they 
would like more interactions with students from other daily 
groups. 

More than one-third of students found working with 
their peers and undergraduate group leaders to be as 
beneficial as interacting with the faculty advisor or graduate 
student mentors. One first-year participant summed up their 
different roles best by writing that the faculty advisor "taught 
me a lot of things about radar, and my team taught me 
things they learned in more advanced classes." Another 
multiyear participant reported that all interactions were 
helpful but thought "students learned the most from 
graduate students coming in and quizzing us because they 
were seen as an all-knowing person, while the group leaders 
were seen more as friends." This mentality may partially 
explain why multiyear participants were more likely to find 
their interactions with the faculty advisor and graduate 
student mentor most beneficial, particularly in discussing 
graduate school or career options. Multiyear students were 
also more likely to provide high praise for the graduate 
student mentors and empathize with the demand on the 
faculty advisor's time, as indicated by a student who wanted 
more interaction with the faculty advisor but realized "this is 
not always practical." 

Allowing advanced undergraduates to direct less expe¬ 
rienced students and assist in the implementation and 
selection of new program components and research 
objectives also enabled students to claim some ownership 
of SOAP. Many students, particularly first-year participants, 
indicated that peer interactions were as beneficial as those 
with the faculty advisor and graduate student mentors. 
Students also coined a program mascot (a rubber chicken 
appropriately named Rossby after Carl-Gustaf Rossby), 


developed the SOAP logo and Web site, and often 
volunteered for extra responsibilities (e.g., meeting at 5 
a.m. to launch special radiosondes and overnight radar 
operations). All of these factors likely contributed to SOAP's 
65% retention rate of nongraduating students and many 
graduating students declaring SOAP as either one of the 
experiences or the most important experience they had at 
TAMU on departmental exit surveys. 

IMPLICATIONS AND LESSONS LEARNED 

This article evaluates undergraduate experiences and 
impacts from participating in an atmospheric science 
research and education program (SOAP) and describes the 
structure, components and implementation of this 1-credit- 
hour research course at TAMU. Analyzing student surveys 
and reflecting upon implementing SOAP supports the 
following conclusions, many of which should apply to all 
geosciences instead of being unique to atmospheric science: 

• Although specific implementation strategies will 
depend on individual resources and objectives, faculty 
beginning new research and education programs 
should establish an organizational framework with 
clear expectations that their resources and time will 
allow them to sustain. Faculty should give students 
autonomy in completing data collection and analysis 
tasks that all levels of students may perform with 
appropriate training, solicit their feedback, and 
encourage them to pursue more advanced research 
projects. 

• Students who participated in SOAP for multiple years 
were significantly more confident in performing 
program tasks than single-year participants and were 
significantly more likely to obtain meteorology or 
science-related employment upon graduation (94% 
versus 69%). Although environmental and involve¬ 
ment variables external to SOAP may partially 
account for these results, they still suggest that 
research and education programs have a strong 
influence on students' career outcomes and fostering 
positive self-efficacy. 

• Many students (41%) indicated that their most 
beneficial interactions were with the faculty advisor 
or graduate student mentors, but first-year partici¬ 
pants were significantly more likely to indicate that 
their interactions with undergraduate student leaders 
or peers were most beneficial. Interactions with the 
faculty advisor or graduate student mentors were 
more important than or equal in importance to peer 
interactions for almost all returning students. 


TABLE VIII: Most beneficial interactions stated by SOAP participants from 2007-2010. 1 


Most Beneficial Interactions per Years Participation 

1 y (n = 66) 

2 y (n = 32) 

3-4 y (n = 16) 

All (n = 114) 

Faculty advisor or graduate student mentors 

36% 

44% 

56% 

41% 

Undergraduate group leaders or peers 

27% 

9% 

— 

19% 

Both (undergraduates and graduate students or faculty) 

32% 

41% 

38% 

35% 

No response 

5% 

6% 

6% 

5% 


1 Students responded to the question, "SOAP incorporates interaction with your peers, group leaders, grad students, and the faculty advisor. Which 
interaction or interactions were most beneficial to you?" Some multiyear participants are reflected in multiple categories depending on which year they 
took the survey. 
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Experiences in implementing SOAP provided many 
lessons that investigators proposing new research and 
education programs may also consider. First, figuring out a 
way to continue the program in some form after the initial 
funding period expires is critical. Students were concerned 
about the void that would be left in the meteorology 
curriculum after SOAP ended in 2010, prompting other 
faculty in the Department of Atmospheric Sciences at TAMU 
to implement the original SOAP paradigm into follow-up 
projects like summer SOAP and the Student Operational 
Upper-Air Project (SOUP). Students in summer SOAP 
(TAMU College of Geosciences, 2013) learn radar, upper- 
air, remote, and in situ observation fundamentals and carry 
out follow-up investigations during summer semesters. 
SOUP students launch weather balloon soundings year- 
round during severe and hazardous weather situations and 
subsequently send their data to the National Weather 
Service to assist their forecasts. Both programs incorporate 
peer-to-peer interactions and accommodate up to 25 
students overseen by one or two faculty and graduate 
student mentors. Funding has come from the College of 
Geosciences at TAMU to enhance high-impact learning by 
undergraduates, the National Aeronautics and Space Ad¬ 
ministration New Investigator Program, and the National 
Oceanic and Atmospheric Administration and Cooperative 
Program for Operational Meteorology, Education, and 
Training (NOAA-COMET) Partners Project. 

Second, undergraduate group leaders, graduate student 
mentors, and the faculty advisor in SOAP had to consider 
their participants' capabilities, figuring out how to teach 
meteorological concepts without formal pedagogical train¬ 
ing. This study suggests that mentors should be cognizant of 
their students' confidence in performing research, particu¬ 
larly for students involved in research for the first time who 
will likely be less confident than students with prior research 
experience. Balancing the number of novice and experienced 
students in each daily group while maintaining the regular 
presence of faculty and graduate student mentors typically 
ensured that struggling first-year SOAP students received 
greater assistance. However, administering ungraded as¬ 
sessments before, during, and after the program would 
better identify struggling students and present an opportu¬ 
nity to evaluate student learning for the program as a whole. 

Third, the lack of individual assessments and surveys 
before and during the program made evaluating students' 
capabilities and performance in SOAP and attributing its 
effect on their career goals and outcomes more subjective 
than is preferred. In particular, we recommend administer¬ 
ing longitudinal surveys before the program to better 
understand students' baseline differences in environmental 
and involvement factors, career goals, and general confi¬ 
dence, interest, and experience levels. If this is not practical, 
studying a valid comparison group of undergraduates with 
reliable datasets would help investigators better distinguish 
between student gains and changes caused by factors within 
and outside their program. However, content assessments 
should be minimized, because many students joined SOAP 
to gain hands-on learning experiences outside of class and 
as one student wrote, "learn without the stresses of exams." 

Finally, although we had the foresight to assess student- 
reported experiences in SOAP, these surveys were primarily 
administered to improve the program and evaluate its 


impacts on students as opposed to investigating educational 
research questions. Libarkin and Kurdziel (2001) outline 
methods for carrying out deliberate educational research 
similar to the scientific method, with special emphasis on 
considering how assessments or surveys will be adminis¬ 
tered (including when and to whom) and analyzed. 
Charlevoix (2008) suggests that higher education may be 
required to provide evidence of student learning in the near 
future; if so, a strong base of scholarly research on teaching 
and learning is a prerequisite. She notes that Texas has 
already considered legislation to mandate testing require¬ 
ments for graduating college seniors that undoubtedly 
would originate within individual departments, many of 
which are likely not well prepared for providing evidence of 
student learning and achievement. In addition, some 
funding agencies that already mandate an educational 
component for some proposals may call for more rigorous 
and deliberate educational research plans in the future. 
Therefore, sharing and properly assessing teaching and 
learning in traditional courses and other learning experienc¬ 
es would help universities proactively prepare for such 
changes while continuing to grow the knowledge base of 
teaching and learning in the geosciences. 
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